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Abstract 

Intelligence  analysts  engage  in  information  seeking,  evaluation,  prediction, 
and  reporting  behavior  in  an  extremely  information-intensive  work  environ¬ 
ment.  A  Cognitive  Task  Analysis  (CTA)  was  conducted  on  intelligence 
analysts  to  capture  data  that  will  provide  input  to  support  development  of  a 
computational  model  of  the  analyst's  processes  and  analytic  strategies.  A 
hybrid  method  was  used  to  conduct  the  CTA,  including  a  modified  version  of 
the  critical  decision  method.  Participants  were  asked  to  describe  an  example 
of  a  critical  analysis  assignment  where  they  had  to  collect,  analyze,  and 
produce  a  report  on  intelligence  of  a  strategic  nature.  Procedures  used  to 
conduct  the  CTA  are  described  in  this  paper  along  with  initial  results.  Several 
factors  contribute  to  making  the  analyst's  task  challenging:  (7)  time  pressure, 

(if)  a  high  cognitive  workload,  and  (Hi)  difficult  human  judgments.  Human 
judgments  are  involved  in  considering  the  plausibility  of  information, 
deciding  what  information  to  trust,  and  determining  how  much  weight  to  place 
on  specific  pieces  of  data.  Intelligence  analysis  involves  a  complex  process  of 
assessing  the  reliability  of  information  from  a  wide  variety  of  sources  and 
combining  seemingly  unrelated  events.  This  problem  is  challenging  because  it 
involves  aspects  of  data  mining,  data  correlation  and  human  judgment. 

INTRODUCTION 

Intelligence  analysis  involves  a  complex  process  of  assessing  the  reliability  of 
information  from  a  wide  variety  of  sources  and  combining  seemingly  unrelated  events. 
Sorting  through  huge  amounts  of  information,  produced  from  a  variety  of  sources  (i.e., 
human,  electronic,  open  source,  measures  and  signals  intelligence,  and  imagery),  and 
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represented  in  many  different  forms  (e.g.,  written  and  oral  reports,  satellite  images,  tables 
of  numeric  data)  to  construct  an  accurate  depiction  of  a  situation  and  make  predictions 
regarding  the  situation  presents  a  difficult  problem  for  the  intelligence  analyst  (IA). 
Products  of  this  information  foraging  and  analysis  are  used  by  senior  decision-makers  to 
make  high-stakes  decisions.  The  problem  faced  by  intelligence  analysts  is  challenging 
because  it  involves  aspects  of  data  mining,  data  correlation  and  human  judgment  (NIMD, 
2002). 

Intelligence  analysis  is  a  dynamic  and  highly  iterative  process  that  involves  viewing  the 
information  from  different  viewpoints  in  order  to  examine  competing  hypotheses  or 
develop  an  understanding  of  a  complex  issue.  The  critical  role  of  the  human  is  in  adding 
"value"  to  original  inputs  by  integrating  disparate  information  and  providing  an 
interpretation  (Krizen,  1999).  This  integration  and  interpretation  entails  difficult,  complex 
judgments  to  make  sense  of  the  information  obtained. 

This  research  effort  seeks  to  develop  analytic  models  of  the  intelligence  analyst  that  will 
ultimately  be  used  to  develop  computational  models  of  tasks  performed  by  intelligence 
analysts.  The  goal  is  to  develop  a  prototype  system  to  aid  intelligence  analysts  (IAs) 
through  the  use  of  novel  human-information  interaction  techniques  and  study  the  impact 
of  these  techniques  on  performance  and  learning  in  intelligence  tasks.  Information 
foraging  theory  (Pirolli  and  Card,  1998;  Pirolli  and  Card,  1999)  is  being  applied  in  this 
research  on  tasks  that  involve  information-intensive  work  where  the  approach  is  to 
analyze  the  tasks  as  an  attempt  by  the  user  to  "maximize  information  gained  per  unit 
time.”  Models  developed  as  a  result  of  this  research  will  be  used  to  help  intelligence 
analysts  in  locating  and  gathering  information  from  large  collections  of  information, 
synthesizing  and  developing  an  understanding  of  this  information,  and  producing  some 
product,  such  as  a  briefing  or  actionable  decision. 

Techniques  to  Enhance  Processing  of  Intelligence  Data 

Recent  world  events  have  focused  attention  on  some  of  the  inherent  challenges  involved 
in  performing  intelligence  analysis.  As  a  result,  increased  research  is  being  conducted  to 
develop  new  training,  tools,  and  techniques  that  will  enhance  the  processing  of 
intelligence  data.  For  example,  support  and  training  in  the  organizing  and  piecing 
together  aspects  of  intelligence  analysis  and  decision  making  has  been  identified  by  the 
Assistant  Secretary  of  Defense  for  Networks  and  Information  Integration  (OASD/NII) 
Research  Program  as  an  area  that  is  greatly  in  need  of  more  basic  and  applied  research. 
One  current  research  thread  that  seeks  to  address  this  need  is  the  Novel  Information  from 
Massive  Data  (NIMD)  program  where  the  goal  is  to  develop  an  “information  manager”  to 
assist  analysts  in  dealing  with  the  high  volumes  and  disparate  types  of  data  that  inundate 
intelligence  analysts.  The  NIMD  research  program  seeks  to  develop  techniques  that 
“structure  data  repositories  to  aid  in  revealing  and  interpreting  novel  contents"  and 
techniques  that  can  accurately  model  and  draw  inferences  about  (1)  rare  events  and  (2) 
sequences  of  events  (widely  and  sparsely  distributed  over  time). 

Another  related  research  thread  involves  constructing  detailed  models  of  users  interacting 
with  the  World  Wide  Web  (WWW),  where  the  ultimate  goal  is  to  develop  intelligent 
agents  that  can  perform  data  mining  in  support  of  the  intelligence  mining  process.  A 
large  source  of  information  that  is  searched  through  by  IAs  for  this  task  involves  open- 
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source  material  (Connable,  2001).  Pirolli,  Fu,  Reeder,  and  Card  (2002)  have  developed  a 
methodology  for  studying  and  analyzing  ecologically  valid  WWW  tasks.  A  user  trace  is 
created  of  all  significant  states  and  events  in  the  user- WWW  interaction  based  on 
analysis  of  eye  tracking  data,  application-level  logs,  and  think-aloud  protocols.  These 
researchers  have  applied  this  user-tracing  architecture  to  develop  simulation  models  of 
user- WWW  interaction  and  to  compare  simulation  models  against  user-trace  data.  When 
the  same  tasks  as  were  presented  to  the  observed  users  are  presented  to  the  simulation 
model,  SNIF-ACT  (Scent-based  Navigation  and  Information  Foraging  in  the  ACT 
theory),  the  model  simulates  activity  with  the  WWW  to  conduct  the  tasks. 

Each  action  of  the  SNIF-ACT  simulation  is  directly  compared  with  observed  user  actions 
via  the  user-tracing  architecture  (Card,  et  al,  2001).  This  user-modeling  research  is  being 
extended  to  develop  computational  models  of  intelligence  analysts  interacting  with  the 
WWW.  Information  foraging  theory  (Pirolli  and  Card,  1998;  Pirolli  and  Card,  1999)  is 
being  applied  in  this  research  on  tasks  that  involve  information-intensive  work  where  the 
approach  is  to  analyze  the  tasks  as  an  attempt  by  the  user  to  maximize  information  gained 
per  unit  time. 

The  goal  for  the  research  described  in  this  paper  is  to  conduct  a  Cognitive  Task  Analysis 
(CTA)  to  collect  the  type  of  data  that  will  support  development  of  computational  models 
of  the  intelligence  analyst's  processes,  biases,  and  analytic  strategies.  The  critical  decision 
method  (Klein  Calderwood,  &  Macgregor,  1989)  was  modified  to  develop  domain- 
specific  cognitive  probes  that  elicit  information  on  how  analysts  obtain  and  use 
information,  schemas  employed  to  conceptualize  the  information,  hypotheses  developed 
to  analyze  this  information,  and  products  developed  as  a  result  of  their  analysis. 

Cognitive  Task  Analysis 

Our  goal  for  conducting  a  CTA  is  to  capture  data  that  will  provide  input  to  support 
development  of  a  computational  model  of  the  intelligence  analyst's  processes  and  analytic 
strategies.  CTA  is  an  extension  of  traditional  task  analysis  techniques  to  produce 
information  regarding  the  knowledge,  cognitive  strategies,  and  goal  structures  that 
provide  the  foundation  for  task  performance  (Chipman,  Schraagen,  and  Shalin,  2000). 
The  goal  of  CTA  is  to  discover  the  cognitive  activities  that  are  required  for  performing  a 
task  in  a  particular  domain  to  identify  opportunities  to  improve  performance  by  providing 
improved  support  of  these  activities  (Potter,  Roth,  Woods,  and  Elm,  2000). 

A  detailed,  accurate  cognitive  model  that  delineates  the  essential  procedural  and 
declarative  knowledge  is  necessary  to  develop  effective  training  procedures  and  systems 
(Annett,  2000).  This  entails  building  a  model  that  captures  the  analysts'  understanding  of 
the  demands  of  the  domain,  the  knowledge  and  strategies  of  domain  practitioners,  and 
how  existing  artifacts  influence  performance.  CTA  can  be  viewed  as  a  problem-solving 
process  where  the  questions  posed  to  the  subject-matter  experts,  and  the  data  collected, 
are  tailored  to  produce  answers  to  the  research  questions,  such  as  training  needs  and  how 
these  training  problems  might  be  solved  (DuBois  &  Shalin,  2000). 

Use  of  multiple  techniques 
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Analysis  of  a  complex  cognitive  task,  such  the  intelligence  analyst's  job,  often  requires 
the  use  of  multiple  techniques.  When  results  from  several  techniques  converge 
confidence  is  increased  regarding  the  accuracy  of  the  CTA  model  (Cooke,  1994; 
Hoffman,  Shadbolt,  Burton,  &  Klein,  1995;  Potter,  et  al,  2000).  During  the  initial 
bootstrapping  phase  of  this  research,  several  CTA  approaches  were  examined  with  an  eye 
toward  determining  which  approach  would  be  most  productive  for  our  domain  of  interest. 

Applied  Cognitive  Task  Analysis.  The  Applied  Cognitive  Task  Analysis  (ACTA) 
Method  uses  interview  techniques  to  elicit  information  about  the  tasks  performed  and 
provides  tools  for  representing  knowledge  (Militello  and  Hutton,  1998).  Discovery  of  the 
difficult  job  elements,  understanding  expert  strategies  for  effective  performance,  and 
identification  of  errors  that  a  novice  might  make  are  objectives  for  using  the  ACTA 
method.  The  focus  for  researchers  using  the  ACTA  method  is  on  interviews  where 
domain  practitioners  describe  critical  incidents  they  have  experienced  while  engaged  in 
their  tasks  and  aspects  of  the  task  that  made  the  task  difficult. 

Many  CTA  techniques  were  developed  and  used  for  tasks  that  involve  the  practitioner 
making  decisions  and  taking  a  course  of  action  based  on  these  decisions,  e.g.,  firefighters, 
tank  platoon  leaders,  structural  engineers,  paramedics,  and  computer  programmers.  One 
of  the  goals  for  many  CTA  techniques  is  to  elicit  information  on  actions  taken  and  the 
decisions  leading  up  to  those  actions  is.  However,  the  scenarios  encountered  during  the 
IA's  job  do  not  fit  this  typical  pattern  because  making  a  decision,  and  taking  action/s 
based  on  these  decisions,  is  not  the  immediate  goal.  One  finding  that  emerged  during  the 
initial  phase  of  this  research  was  that  making  decisions  is  not  a  typical  part  of  the  IA’s 
task.  Instead,  the  major  tasks  consist  of  searching  through  vast  amounts  of  data  and 
information  to  filter,  synthesize,  and  correlate  the  information  to  produce  a  report 
summarizing  what  is  known  about  a  particular  situation  or  state  of  affairs.  Then,  the 
person  for  whom  the  report  is  produced  takes  actions  based  upon  the  information 
contained  in  the  report. 

Our  use  of  the  ACTA  method  produced  valuable  data  for  the  initial  bootstrapping  phase 
of  this  research  where  the  goal  was  to  learn  about  the  task,  the  cognitive  challenges 
associated  with  task  performance,  and  to  determine  what  tasks  to  focus  on  during  ensuing 
phases  of  the  CTA  research.  A  Knowledge  Audit  and  a  Cognitive  Demands  Table  are 
typically  produced  when  using  the  ACTA  method.  (Table  1,  in  the  Results  section, 
presents  an  example  of  one  Cognitive  Demands  Table  that  was  produced  as  a  result  of 
using  the  ACTA  method.)  During  phase  two  we  used  a  different  method  during  our 
interviews  to  capture  the  essence  of  the  IA's  job.  because  the  nature  of  the  IAs  task  places 
greater  emphasis  on  deductive  reasoning,  looking  for  patterns  of  activity,  and  making 
judgments  about  the  level  of  risk  present  in  a  particular  situation. 

Critical  Decision  Method.  The  Critical  Decision  Method  (CDM)  is  a  semi-structured 
interview  technique  developed  to  obtain  information  about  decisions  made  by 
practitioners  when  performing  their  tasks.  Specific  probe  questions  help  experts  describe 
what  their  task  entails.  CDM's  emphasis  on  non-routine  or  difficult  incidents  produces  a 
rich  source  of  data  about  the  performance  of  highly  skilled  personnel  (Klein, 
Calderwood,  &  Macgregor,  1989).  By  focusing  on  critical  incidents  this  method  is  also 
efficient  as  it  aids  in  uncovering  elements  of  expertise  that  might  not  be  found  in  routine 
incidents  (Klein  &  Hoffman,  1993).  An  emphasis  on  non-routine  cases  helps  to  ensure  a 
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comprehensive  coverage  of  the  subject  matter.  In  this  paper  we  describe  the  development 
and  use  of  a  modified  version  of  the  CDM  and  preliminary  results  derived.  The 
distinction  is  that  we  asked  the  study  participants  to  describe  an  example  of  a  strategic 
analysis  problem  in  lieu  of  a  critical  decision  problem.  This  method  was  modified  to 
develop  cognitive  probes  that  elicit  information  on  how  analysts  obtain  and  use 
information,  schemas  employed  to  conceptualize  the  information,  hypotheses  developed 
to  analyze  this  information,  and  products  developed  as  a  result  of  their  analysis. 

METHOD 

Procedures  developed  and  used  to  conduct  a  modified  version  of  the  CDM  are  described 
in  the  following  section.  Interview  probes  were  developed  to  ask  participants  to  describe 
an  example  of  a  strategic  analysis  problem  versus  a  critical  decision  problem. 

Participants 

Ten  U.S.  Navy  officer-students,  currently  enrolled  in  a  graduate  school  program  at  the 
Naval  Postgraduate  School  (NPS),  Monterey,  CA,  were  interviewed.  Participants  were 
contacted  via  e-mail  with  the  endorsement  of  their  curriculum  chair  and  were  asked  to 
volunteer  for  this  study.  These  officers  (0-3  -  0-4)  were  students  in  either  the  Intelligence 
Information  Management  or  the  National  Security  Affairs  curricula  at  NPS.  Participants 
had  an  average  of  ten  years  experience  working  as  intelligence  analysts.  Thus,  these 
participants  were  considered  experts  as  the  literature  generally  defines  an  "expert"  as  an 
individual  who  has  over  ten  years  experience  and  "would  be  recognized  as  having 
achieved  proficiency  in  their  domain"  (Klein,  et  al,  1989,  p.  462). 

Procedure 

During  phase  one  the  objective  was  to  begin  studying  the  job  to  determine  what  tasks 
merit  the  detailed  attention  of  a  CTA  and  to  identify  a  representative  set  of  problems  or 
cases.  Information  gathered  during  this  phase  served  as  an  advance  organizer  by 
providing  an  overview  of  the  task  and  helped  to  identify  the  cognitively  complex 
elements  of  the  task. 

Phase  One.  Semi-structured  interviews  were  conducted  where  intelligence  analysts  were 
asked  to  recall  an  incident  from  past  experience.  Analysts  were  asked  to  identify  exam¬ 
ples  of  the  challenging  aspects  of  their  tasks  and  why  these  tasks  are  challenging,  the  cues 
and  strategies  that  are  used  by  practitioners,  and  to  describe  the  context  of  the  work. 
Interviews  were  scheduled  for  one  and  one-half  hours  at  a  time  that  was  convenient  for 
each  participant.  Three  interviewers  were  present  for  each  of  the  first  six  interviews.  The 
interviews  were  tape-recorded  and  transcribed  and  the  analysis  was  performed  using  the 
transcription  and  any  other  materials  produced  during  the  interview,  e.g.,  task  diagrams. 
The  focus  was  on  discovery  of  the  difficult  job  elements,  understanding  expert  strategies 
for  effective  performance,  and  identification  of  errors  that  a  novice  might  make. 

Domain  experts  were  asked  to  describe  critical  incidents  they  had  experienced  on  their 
job  and  aspects  of  the  task  that  made  the  task  difficult.  The  first  group  of  intelligence 
analysts  had  a  variety  of  assignments  in  their  careers,  however  the  majority  of  their 
experience  was  focused  on  performing  analysis  at  the  tactical  level.  (Tactical  level 
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analysis  refers  to  analysis  of  information  that  will  impact  mission  performance  within  the 
local  operating  area,  e.g.,  of  the  battle  group,  and  generally  within  the  next  24  hours.) 
During  this  bootstrapping  phase  of  our  CTA  effort,  we  learned  that  there  are  several 
career  paths  for  intelligence  analysts.  These  career  paths  can  be  categorized  as  either 
having  more  of  a  technology  emphasis  where  the  focus  is  on  systems,  equipment,  and 
managing  the  personnel  who  operate  and  maintain  this  equipment  or  an  analytical 
emphasis  where  the  focus  and  experience  is  on  performing  long-range  analysis. 

The  ACTA  method  produced  valuable  data  for  the  initial  phase  of  this  research  where  the 
goal  was  to  learn  about  the  task,  the  cognitive  challenges  associated,  and  determine  what 
tasks  to  focus  on  during  ensuing  phases  of  the  CTA  research.  After  analyzing  the  data 
from  the  initial  set  of  interviews,  we  determined  that  we  needed  to  broaden  the  set  of 
interview  probes  to  uncover  the  bigger  picture  of  how  intelligence  analysts  approach 
performing  their  job.  Descriptions  of  experiences  at  the  tactical  level  did  not  provide 
examples  of  the  types  of  problems  or  cases  that  could  benefit  from  the  training 
technology  envisioned  as  the  ultimate  goal  for  this  research.  For  the  second  group  of 
interviews,  interviewees  were  drawn  from  the  National  Security  Affairs  Department 
where  there  is  stronger  analytical  emphasis  in  the  curriculum  and  the  analysts  have  had 
experience  with  analysis  assignments  at  the  strategic  level.  This  second  group  of 
participants  was  very  articulate  in  describing  assignments  where  they  performed  analysis 
of  critical  topics  at  the  strategic  level. 

Phase  Two.  During  this  second  phase  a  structured  set  of  domain-specific  interview 
probes  was  developed  specifically  for  use  with  this  group  of  participants.  One  interviewer 
conducted  the  initial  interviews;  each  interview  lasted  approximately  one  and  one-half 
hours.  Once  the  initial  interview  was  analyzed,  the  participant  was  asked  to  return  for  a 
follow-up  interview.  All  three  interviewers  were  present  for  the  follow-up  interviews 
with  this  second  group  of  intelligence  analysts. 

Since  the  major  tasks  for  I A  consist  of  searching  through  vast  amounts  of  information  to 
filter,  synthesize,  and  summarize  what  is  known  about  a  particular  situation  or  state  of 
affairs  interview  probe  questions  provided  in  the  literature  were  modified  to  capture 
information  about  their  approach  to  gathering  and  analyzing  information.  Probes  were 
developed  to  focus  the  discussion  on  a  critical  analysis  assignment  where  the  analyst  had 
to  produce  a  report  on  intelligence  of  a  strategic  nature.  Interview  probes  were  developed 
to  capture  information  on  the  types  of  information  used,  how  this  information  was 
obtained,  and  the  strategies  used  to  analyze  this  information. 

Modified  Critical  Decision  Method 

A  modified  version  of  the  critical  decision  method  (CDM)  was  developed  and  used  for 
this  task  domain.  Interview  probe  questions  provided  in  the  literature  (Hoffman,  Coffey, 
and  Ford,  2000)  were  modified  to  focus  the  discussion  on  a  critical  incident  or 
assignment  where  the  analyst  had  to  collect,  analyze,  and  produce  a  report  on  intelligence 
of  a  strategic  nature.  Examples  of  such  strategic  analysis  problems  might  include 
assessments  of  the  capabilities  of  nations  or  terrorist  groups  to  obtain  or  produce  weapons 
of  mass  destruction,  terrorism,  strategic  surprise,  political  policy,  or  military  policy. 
Participants  were  asked  to  describe  what  they  did  step-by-step  and  were  asked  to 
construct  a  timeline  to  illustrate  the  entire  analysis  process. 
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Deepening  Probes.  Domain-specific  cognitive  probes  were  developed  to  capture 
information  on  the  types  of  information  the  IA  was  seeking,  the  types  of  questions  the 
analyst  was  asking,  and  how  this  information  was  obtained.  Additional  information  was 
collected  on  mental  models  used  by  analysts,  hypotheses  formulated  and  the  types  of 
products  that  are  produced.  Table  1  lists  the  questions  posed  to  the  participants  during 
the  initial  interview.  Topics  for  which  participants  conducted  their  analyses  included 
modernization  of  a  particular  country's  military,  whether  there  would  be  a  coup  in  the 
Philippines  and  the  potential  impact  on  the  US  if  there  was  a  coup,  and  the  role  for  the 
newly  created  Department  of  Homeland  Security. 

Table  1.  Modified  Critical  Decision  Method:  Deepening  Probes 


Probe  Topic 

Probe 

Information 

What  information  were  you  seeking,  or  what  questions  were 
you  asking? 

Why  did  you  need  this  information? 

How  did  you  get  that  information? 

Were  there  any  difficulties  in  getting  the  information  you 
needed  from  that  source? 

What  was  the  volume  of  information  that  you  had  to  deal  with? 

What  did  you  do  with  this  information? 

Would  some  other  information  been  helpful? 

Mental  Models/ 
Schemas 

As  you  went  through  the  process  of  analysis  and  understanding 
did  you  build  a  conceptual  model? 

Did  you  try  to  imagine  important  events  over  time? 

Did  you  try  to  understand  important  actors  and  their 
relationships? 

Did  you  make  a  spatial  picture  in  your  head? 

Can  you  draw  me  an  example  of  what  it  looks  like? 

Hypotheses 

Did  you  formulate  any  hypotheses? 

Did  you  consider  alternatives  to  those  hypotheses? 

Did  the  hypotheses  revise  your  plans  for  collecting  and 
marshalling  more  information?  If  so,  how? 

Intermediate 

Products 

Did  you  write  any  intermediate  notes  or  sketches? 

Follow-up  Probes.  Once  the  data  from  the  initial  interviews  was  transcribed  and 
analyzed,  participants  were  asked  to  return  for  a  follow-up  interview.  The  goal  during 
this  session  was  to  refine  our  understanding  of  the  IA's  task.  The  analyst  was  asked  to 
review  the  timeline  produced  during  the  first  interview  session  and  to  elaborate  on  the 
procedures  and  cognitive  strategies  employed.  Probes  used  during  the  follow-up 
interview  are  listed  in  Table  2. 
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Probes  included  questions  about  the  participants'  goals,  whether  this  analysis  was  similar 
to  other  analysis  assignments,  use  of  analogues,  and  whether  hypotheses  were  formed. 
Other  probes  asked  about  the  types  of  questions  raised  during  their  analysis,  methods 
used,  information  cues  they  used  to  seek  and  collate  information,  and  the  types  of  tools, 
e.g.,  computer  software,  they  used  to  perform  their  analysis.  During  this  second  interview 
we  went  through  the  same  intelligence  analysis  problem  with  the  goal  of  obtaining 
additional  details  to  refine  our  understanding  of  the  entire  analysis  process.  This  included 
the  types  of  information  they  used,  and  how  they  structured  their  analysis  to  answer  the 
strategic  question  they  had  been  assigned. 

Table  2.  Follow-up  Probes  Used  for  Modified  Critical  Decision  Method 


Probe  Topic 

Probes 

Goals 

What  were  your  specific  goals  at  the  time? 

Standard 

Does  this  case  fit  a  standard  or  typical  scenario? 

Scenarios 

Does  it  fit  a  scenario  you  were  trained  to  deal  with? 

Analogues 

Did  this  case  remind  you  of  any  previous  case  or 
experience? 

Hypotheses  and 

What  hypotheses  did  you  have? 

Questions 

What  questions  were  raised  by  that  hypothesis? 

What  alternative  hypotheses  did  you  consider? 

What  questions  were  raised  by  that  alternative  hypothesis? 

Information  Cues 

As  you  collected  and  read  information,  what  things  triggered 

for  Hypotheses  and 
Questions 

questions  or  hypotheses  that  you  later  followed  up? 

Information  Tools 

What  sort  of  tools,  such  as  computer  applications,  did  you 
use? 

What  information  source  did  you  use? 

What  difficulties  did  you  have? 

RESULTS 

A  description  of  what  has  been  learned  during  the  first  two  phases  of  this  CTA  research 
with  intelligence  analysts  is  presented  in  this  section.  These  results  are  based  on  analysis 
of  data  obtained  during  both  phases  of  this  research,  i.e.,  using  the  ACTA  method  and  the 
modified  CDM  method.  During  phase  one,  participants  focused  on  providing  descriptions 
of  the  cognitively  challenging  aspects  of  the  task;  often  the  discussion  was  focused  on 
developing  a  product  to  support  operations  at  the  tactical  level.  (The  tactical  level 
generally  refers  to  operations  that  will  occur  within  hours  or  days.)  During  phase  two, 
the  emphasis  was  on  describing  tasks  where  the  focus  was  on  analysis  of  intelligence  in 
order  to  produce  a  report  to  answer  a  question  of  interest  at  the  strategic  level.  The  length 
of  time  participants  in  phase  two  had  devoted  to  the  assignments  that  they  described 
ranged  from  six  weeks  to  three  and  one-half  years. 
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Applied  Cognitive  Task  Analysis 


The  initial  set  of  knowledge  representations  for  the  IA’s  job  (produced  using  the  ACTA 
method)  provided  the  basis  for  the  more  detailed  CTA.  Table  3  presents  an  example  of 
the  one  of  the  tables  that  can  be  produced  using  the  ACTA  method.  The  Cognitive 
Demands  Table  was  produced  based  on  analysis  of  data  captured  during  an  interview 
with  one  participant  during  the  first  phase  of  this  CTA  research. 


Table  3.  Cognitive  Demands  Table:  NPS#2 


Cognitive  Why  Difficult 
Demand 

Cues 

Strategies 

Potential 

Errors 

Synthe-  • 

Lack  of  technical 

Difficult  to  know 

Emphasize  type  of 

•  Potential  for  biases 

sizing 

familiarity  with  different 

how  to  weight 

data  analyst  has 

•  Tendency  to  focus  on 

data 

types  of  data 

different  kinds  of 

experience  with, 

type  of  data  analyst 

• 

Domain  expertise  is 
needed  to  analyze 
each  class  of  data 
(HUMINT,SIGINT, 

ELINT,  IMAGERY,  etc.) 

data 

and  disregard 
other  data 

has  experience  with 
and  to  ignore  data  you 
do  not  understand 

Synthe-  • 

No  one  database  exists 

Systems  produce 

Different 

•  Users  develop  comfort 

sizing 

that  can  correlate 

different  "results," 

commands  rely 

level  with  their  system 

data 

across  systems 

e.g.,  mensuration 

on  different 

and  its  associated 

• 

No  one  database  can 
correlate  all  inputs 
from  many  different 
analysts  to  form  one 
coherent  picture 

process  produces 
different  latitude/ 
longitude  coordi¬ 
nates  from 
other  systems 

databases  in  which 
they  have  developed 
trust 

database;  this  can  lead 
to  wrong  conclusion 

Synthe-  • 

Databases  are  cumber¬ 

Users  don't  always 

Use  own 

•  Rely  on  trend 

sizing 

Data 

• 

some  to  use:  Poor 
correlation  algorithms 
System  presents  results 
that  users  do  not  trust, 
tracks  are  "out  of  whack." 

understand  infor¬ 
mation  system 
presents.  Too  many 
levels  in  system  are 
not  transparent 

experience 

information 

Notic-  • 

Time  critical  information 

Need  to  decide 

Need  to  rely  on 

•  Refer  to  other 

Ing  data 

• 

is  difficult  to  obtain 

Need  to  assimilate,  verify 
and  disseminate  in  a  short 
time  window 

whether  imagery 
is  current  enough 
to  proceed  with  strike 
How  long  has  it  been 
there? 

other  sources  to 
verify  current 

sources  to  verify 

Cognitive  Challenges 

The  following  paragraphs  describe  the  cognitive  challenges  inherent  in  performing 
intelligence  analysis. 

Time  Pressure 
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The  IA  task  is  made  difficult  by  the  confluence  of  several  cognitive  processing 
requirements.  Time  pressure  to  produce  reports  for  decision-makers  in  shorter  times  is 
becoming  an  increasingly  stressful  requirement  for  analysts  working  at  all  levels,  from 
tactical  through  strategic  levels.  As  an  example  at  the  strategic  level,  one  participant  had 
six  weeks  to  prepare  a  report  on  a  matter  of  strategic  importance;  this  included  time  to 
gather  all  the  necessary  information,  analyze  the  information  collected  from  diverse 
sources  and  produce  the  report.  This  analyst  had  no  background  knowledge  of  this  area 
and  had  to  begin  by  reading  travel  books  and  other  general  information.  The  assignment 
involved  the  question  of  whether  President  Estrada,  of  the  Philippines,  would  be  deposed 
as  President,  and  if  so,  would  there  be  a  coup?  This  assignment  was  to  include  an  analysis 
of  what  the  impact  would  be  on  the  U.S.  As  an  example  of  time  pressure  at  the  tactical 
level,  another  participant  described  how  the  effect  of  timeline  compression  coupled  with 
organizational  constraints  can  sometimes  "channel  thinking"  down  a  specific  path. 

Multiple  Information  Requirements  and  Complex  Judgments 

Multiple  sources  of  disparate  types  of  data  (e.g.,  open  source,  classified,  general 
reference  materials,  embassy  cables,  interviews  with  experts,  military  records,  etc.)  must 
be  combined  to  predict  complex,  dynamic  events.  The  cognitive  challenges  involved  in 
merging  information  from  these  different  sources  can  be  especially  difficult,  e.g.,  how  to 
filter  and  weight  different  kinds  of  data  when  the  analyst  lacks  experience  with  all  the 
different  types  of  data.  These  different  types  of  data  have  varying  degrees  of  validity  and 
reliability  that  must  be  considered.  Moreover,  domain  expertise  is  often  needed  to 
analyze  each  type  of  data.  For  example,  two  analysts  looking  at  the  same  image  may  see 
different  things.  Many  factors  need  to  be  considered  when  interpreting  imagery  data, 
such  as  the  time  of  day  the  image  was  taken,  how  probable  it  is  to  observe  a  certain  thing, 
and  trends  within  the  particular  country.  A  high  level  of  cognitive  workload  is  produced 
when  a  constant  incoming  stream  of  information  must  be  continuously  evaluated,  updated 
and  synthesized.  An  additional  contributor  to  the  high  workload  is  the  labor-intensive 
process  employed  when  analysts  process  data  manually  because  no  one  single  database 
exists  that  can  correlate  across  the  various  types  of  data  that  must  be  assimilated. 

Difficult  human  judgments  are  involved  in  (?)  considering  the  plausibility  of  information, 
(ii)  deciding  what  information  to  trust,  and  (Hi)  determining  how  much  weight  to  give 
specific  pieces  of  data.  One  aspect  of  the  IA  task  that  is  particularly  challenging  involves 
merging  different  types  of  information  when  the  analyst  does  not  have  technical 
familiarity  with  all  these  types  of  information.  Human  intelligence,  electronic  intelli¬ 
gence,  imagery,  open  source  intelligence,  measures  and  signals  intelligence  can  all 
include  spurious  signals  or  inaccurate  information  due  to  the  system  used  or  to  various 
factors  associated  with  the  different  types  of  data.  Analysts  described  situations  where 
they  gave  greater  weight  to  the  types  of  information  they  understood  and  less  weight  to 
less  understood  information.  Analysts  must  also  resolve  discrepancies  across  systems, 
databases,  and  services  when  correlation  algorithms  produce  conflicting  results  or  results 
that  users  do  not  trust. 

High  Cognitive  Load.  Intelligence  analysts  must  perceive,  filter,  analyze,  synthesize 
and  determine  the  relevance  of  a  continual  stream  of  incoming  information.  This  stream 
of  information  often  pertains  to  several  different  situations.  A  critical  part  of  the  analyst's 
task  involves  projecting  future  anticipated  events  and  making  recommendations  regard- 

11 


ing  whether  to  task  intelligence  gathering  sources  to  capture  additional  data.  Analysts 
must  assess,  compare,  and  resolve  conflicting  information,  while  making  difficult 
judgments  and  remembering  the  status  of  several  evolving  situations.  These  cognitive 
tasks  are  interleaved  with  other  requisite  tasks,  such  as  producing  various  reports.  For 
example,  a  request  to  gather  additional  information  will  often  involve  use  of  an  asset  that 
is  in  high  demand.  Re-tasking  an  asset  can  be  costly,  thus,  tradeoffs  must  be  made 
regarding  the  priority  for  use  of  an  asset  for  one  objective  versus  the  potential  gain  in 
information  when  re-tasking  the  asset  to  satisfy  a  new  objective. 

The  sheer  volume  of  information  makes  it  hard  to  process  all  the  data,  yet  currently 
available  technology  is  not  always  effective  in  helping  the  analyst  assimilate  the  huge 
amount  of  information  that  needs  to  be  synthesized.  Databases  exist  but  are  cumbersome 
to  use  due  to  system  design  issues.  Human  judgment  is  entailed  in  deciding  what 
information  to  trust  and  these  judgments  can  be  particularly  difficult  when  they  need  to 
be  made  under  time  pressure.  Each  type  of  data  has  to  be  assessed  to  determine  its 
validity,  reliability,  and  relevance  to  the  particular  event  undergoing  analysis. 

Potential  for  Cognitive  Biases.  The  high  mental  workload  imposed  on  IAs  introduces  a 
potential  for  cognitive  biases  to  influence  interpretation.  The  potential  for  “cognitive 
tunnel  vision”  to  affect  the  analysis  process  is  introduced  by  the  high  cognitive  load  that 
analysts  often  experience.  For  example,  they  may  miss  a  key  piece  of  information  when 
they  become  overly  focused  on  one  particularly  challenging  aspect  of  the  analysis.  Bias 
may  influence  the  analysis  process  when  analysts  attempt  to  reduce  their  cognitive  load 
by  analyzing  data  they  understand  and  discounting  data  with  which  they  have  less 
experience.  Additionally,  discrepancies  regarding  interpretation  may  result  when 
decision-makers  at  different  locations  (e.g.,  on  different  platforms,  different  services)  rely 
on  systems  that  produce  different  results.  Moreover,  the  sheer  volume  of  information 
makes  it  hard  to  process  all  the  data,  yet  no  technology  is  available  that  is  effective  in 
helping  the  analyst  synthesize  all  the  different  types  of  information. 

Schemas 

In  this  example  the  person  described  his  task  of  having  to  build  a  brief  on  a  political 
question  regarding  whether  President  Estrada  would  be  deposed  from  the  Philippines, 
whether  there  would  be  a  coup,  and  if  there  was  a  coup,  what  the  implications  would  be 
for  the  U.S.?  He  was  asked  to  complete  this  analysis  task  within  a  time  span  of  six  weeks. 
Figure  1  depicts  a  high-level  representation  of  this  task.  From  the  initial  search  of  raw 
reports  he  produced  an  initial  profile.  Many  follow-up  phone  calls  and  additional 
searches  were  conducted  to  fill  in  the  gaps  and  elaborate  on  what  was  learned  during  the 
initial  set  of  queries.  This  step  resulted  in  producing  a  large  number  of  individual  word 
files  on  each  political  person  or  key  player.  These  included  biographies  on  approximately 
125  people,  including  insurgency  leaders,  people  in  various  political  groups,  people  with 
ties  to  crime,  etc.  The  information  in  these  files  was  then  grouped  in  various  ways.  He 
developed  a  set  of  questions  to  use  to  work  backwards  to  review  all  the  material  from 
several  different  vantage  points  to  answer  questions,  such  as:  Will  there  be  a  coup?  Will 
it  be  peaceful  or  not?  Will  it  be  backed  by  the  military?  Will  the  vote  proceed,  or  will 
the  military  step  in,  prior  to  the  vote?  What  is  the  most  likely  scenario  to  pan  out? 
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Figure  1.  NSA  Expert  on  "Will  Estrada  Be  Overthrown  in  a  Military  Coup?" 

Figure  2  depicts  the  various  information  sources  that  the  analyst  researched  to  develop  a 
comprehensive  understanding  of  the  issue.  The  analyst  began,  in  week  one,  by  reading 
general  background  information  to  develop  knowledge  on  the  history  and  cultural 
ethnography  of  the  country  and  also  by  examining  prior  Naval  Intelligence  on  the 
previous  history  for  political  turnover  in  the  Philippines.  During  week  two  he  began 
contacting  Intelligence  Centers  and  reading  U.S.  Embassy  cables,  an  important  source  for 
this  particular  topic.  While  this  step  provided  valuable  information  this  material  was  from 
a  secondary  source,  that  is,  it  represented  someone  else's  analysis  of  the  situation.  This 
necessitated  the  analyst  having  to  decide  which  of  these  reports  were  to  be  given  more 
emphasis  and  in  which  ones  he  could  not  place  as  much  confidence. 

One  way  the  analyst  structured  his  analysis  was  by  sorting  people  according  to  whether 
they  were  pro-Estrada  or  anti-Estrada,  which  figures  would  be  likely  to  drop  allegiance  to 
the  constitution?  and  so  on.  Several  people  involved  in  coup  attempts  around  the  time  of 
President  Aquino  were  now  senators.  Voting  records  provided  another  way  to  sort 
people.  Political  figures'  ties  to  certain  newspapers  were  examined  to  determine  which 
camps  they  would  fall  into.  He  attempted  to  determine  what  biases  key  figures  might 
have.  There  were  many  branches  and  sequels,  thus  many  ways  to  sort  the  information. 
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Figure  2.  Information  Foraging:  Dual  Problem  Space 
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Figure  3  depicts  the  information  schema  used  by  for  this  analyst.  Multiple  ways  of 
grouping  people  were  used  to  consider  how  their  allegiance  would  "fall  out"  based  on 
their  various  associations.  For  example,  he  grouped  key  people  in  both  the  military  and 
civilian  sectors  according  to  their  military  associations,  political,  family,  geographic 
region,  and  various  other  associations  to  determine  whether  they  were  pro-Aquino  or 
anti-Aquino.  Other  ways  of  trying  to  ascertain  their  loyalty  involved  examination  of 
other  types  of  affiliations,  for  example,  boards  they  belong  to.  The  analyst  developed 
many  branches  and  sequels  between  people  and  events  in  his  attempt  to  examine  their 
affiliations  from  many  different  vantage  points.  He  also  reviewed  past  coup  attempts  that 
occurred  around  the  time  of  past-President  Aquino  to  examine  how  these  people's 
allegiances  might  develop.  Another  approach  employed  was  to  try  to  ascertain  which 
camp  certain  political  players  would  fall  into  by  various  groups  and  activities  they  were 
associated  with,  e.g.,  certain  newspapers. 
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Figure  3.  Schemas  Used  to  Analyze  the  Intelligence  Problem 
Summary 

Identification  of  an  appropriate  sample  of  problems  or  tasks  is  essential,  particularly  in 
developing  training  techniques,  to  ensure  sufficient  coverage  of  critical  skills  and 
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knowledge.  The  initial  set  of  interviews  was  conducted  to  develop  a  foundation  of 
knowledge  regarding  the  Intelligence  Analysts'  task  domain.  This  first  phase  of  the 
research  served  to  identify  parts  of  the  job  that  require  skilled  judgment  and  evaluation. 
During  the  next  phase  of  this  research,  additional  analytical  evidence  and  empirical  data 
will  be  gathered  to  further  refine  and  verify  the  CTA  model  of  the  intelligence 
information  analyst's  job.  One  goal  for  this  phase  will  be  to  predict  or  envision  the 
impact  the  technology  will  have  on  cognition  for  the  intelligence  analyst.  A  second  goal 
is  to  influence  the  development  process  so  the  new  training  is  useful. 

DISCUSSION 

One  goal  for  this  effort  is  to  capture  data  that  will  provide  input  to  building  models  of  the 
IA  and  analytic  processes  used  to  perform  intelligence  analysis.  The  goal  of  CTA  is  to 
discover  the  cognitive  activities  that  are  required  for  performing  a  task  to  identify 
opportunities  to  improve  performance  by  providing  improved  support  of  these  activities. 
A  detailed,  accurate  cognitive  model  that  delineates  the  essential  procedural  and 
declarative  knowledge  is  necessary  to  develop  effective  training  procedures  and  systems. 
This  entails  building  a  model  that  captures  the  analysts'  understanding  of  the  demands  of 
the  domain,  the  knowledge  and  strategies  of  domain  practitioners,  and  how  existing 
artifacts  influence  performance.  The  major  impact  of  the  work  described  above  will  be  to 
provide  data  that  will  be  used  to  develop  computational  models  of  the  IA  process. 
Models  developed  as  a  result  of  this  research  will  be  used  to  help  analysts  in  locating  and 
gathering  information  from  large  collections  of  information,  synthesizing  and  developing 
an  understanding  of  this  information. 

FUTURE  RESEARCH 

The  next  phase  of  this  research  will  involve  presenting  an  analysis  task  and  observing  the 
IA  while  performing  the  task  in  a  controlled  laboratory  environment.  Data  will  be 
collected  using  an  eye  tracker,  logging  software  that  collects  all  user  actions  with  a 
WWW  browser,  and  video  recordings  of  think-aloud  verbal  protocols  (Card,  et  al.,  2001; 
Pirolli,  Fu,  Reeder,  and  Card,  2002).  Working  within  a  system  development  process,  to 
support  critical  system  design  issues,  additional  data  and  empirical  evidence  will  be 
collected.  The  CTA  process  is  an  iterative  process  that  builds  on  subsequent  design 
activities.  Phase  three  will  continue  to  refine  the  model  by  continued  exploration  of  the 
domain  and  the  way  practitioners  operate  in  the  domain.  Introducing  new  technology 
will  impact  task  performance.  New  tools  and  training  will  impact  the  cognitive  activities 
to  be  performed  and  enable  development  of  new  strategies.  CTA  techniques  that  might 
be  useful  during  this  phase  include  storyboarding,  participatory  design,  rapid  prototype 
evaluations,  and  observations  of  performance  using  various  degrees  of  fidelity. 
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Background:  Intelligence  Analysis 


•  Sort  through  vast  amounts  of  information  to  construct  an  accurate  depiction  of  a 
situation 

-  Products  of  information  foraging  and  analysis  are  used  by  senior  decisionmakers 
to  make  high-stakes  decisions 

•  Complex  process  of  assessing  reliability  of  information  from  wide  variety  of  sources 
and  combining  seemingly  unrelated  events 

-  Involves  data  mining,  data  correlation,  human  judgment 

•  CTA  -  capture  data  to  provide  input  to  support  development  of  a  computational  model 
of  the  lA’s  processes,  biases,  analytic  strategies 

-  New  method  needed:  scenarios  do  not  fit  typical  pattern  of  making  decisions  and  taking 
actions 

-  Many  CTA  techniques  have  goal:  elicit  information  on  actins  taken  and  decisions  leading  up  to 
those  actions 

-  Greater  emphasis  on  deductive  reasoning,  looking  for  patterns  of  activity,  and  making 
judgment  about  the  level  of  risk  present 

•  Needed  to  take  different  tack  during  interviews  to  capture  essence  lA’s  job 

-  Participants  were  asked  to  describe  an  example  of  a  strategic 
analysis  problem 


Cognitive  Task  Analysis  Methodology 


•  Participants:  10  military  officer-students  enrolled  in  NPS  graduate 
school  program 

-  6  Intelligence  Information  Management;  4  National  Security  Affairs 

-  Average  10  years  experience  as  intelligence  analysts 

•  Phase  One:  Semi-structured  interviews  -  Knowledge  Audit  (ACTA) 

-  Identify  challenging  aspects  of  their  jobs,  cues  and  strategies 

-  Needed  broader  set  of  probes  to  uncover  bigger  picture 

•  Phase  Two:  Draws  from  a  variety  of  CTA  methods 

-  Critical  Decision  Method  (Klein,  et  al.;  Hoffman,  et  al.) 

-  Tailored  to  Intelligence  Analysts 

-  Interview  probes  developed  to  focus  on  a  critical  assignment  where 
analyst  had  to  collect,  analyze  and  produce  report  of  strategic  nature 

*  Types  of  information  used,  how  information  was  obtained 
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Critical  Decision  Method 


We  are  interested  in  the  collection,  analysis,  and  reporting  of  intelligence  of  strategic 
importance  (as  opposed  to  tactical  or  operational  intelligence).  Examples  of  such 
strategic  analysis  problems  might  include  assessments  of  the  capabilities  of  nations 
or  terrorist  groups  to  obtain  or  produce  weapons  of  mass  destruction,  terrorism, 
strategic  surprise,  political  policy,  military  policy  etc. 

Please  try  to  recall  a  recent  significant  assignment  in  which  you  had  to  collect, 
analyze,  and  report  on  an  intelligence  problem  of  a  strategic  nature  (for  example,  a 
class  assignment  that  resulted  in  a  paper,  talk,  briefing,  etc.). 

When  was  it? 

Where  (in  what  course)  was  it? 

What  was  the  specific  assignment? 

What  was  the  final  product,  product  length,  and  product  audience? 

How  much  time  was  devoted  to  this  task? 

Over  what  period  of  time? 

What  specific  training  or  experience  did  you  have  that  was  particularly  relevant  to 
accomplishing  this  assignment? 

What  did  you  do  step  by  step?  (use  as  much  space  as  needed  to  construct  timeline) 
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CDM  -  Deepening 


fa 


Information 

What  information  were  you  seeking,  or  what  questions  were  you 
asking? 

Why  did  you  need  this  information? 

How  did  you  get  that  information?  What  was  the  information 
source? 

Are  there  any  difficulties  in  getting  the  information  you  needed 
from  that  source? 

What  was  the  volume  of  information  that  you  had  to  deal  with? 
What  did  you  do  with  this  information? 

Would  some  other  information  been  helpful? 

Mental 

models/Schemas 

As  you  went  through  the  process  of  analysis  and  understanding 
did  you  build  a  conceptual  model? 

Did  you  try  to  imagine  important  events  over  time? 

Did  you  try  to  understand  important  actors  and  their 
relationships? 

Did  you  make  a  spatial  picture  in  your  head? 

Can  you  draw  me  an  example  of  what  it  looks  like? 

Hypotheses 

Did  you  formulate  any  hypotheses? 

Did  you  consider  alternatives  to  those  hypotheses? 

Did  the  hypotheses  revise  your  plans  for  collecting  and 
marshalling  more  information?  If  so,  how? 

Intermediate  products 

Did  you  write  any  intermediate  notes  or  sketches? 
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CDM  -  Follow-up 


In  the  last  interview,  you  provided  us  with  an  interesting  example  of  an 
analysis  that  you  had  performed.  In  todayO  interview  we  would  like  to 
go  th  rough  that  analysis  again  to  verify  and  elaborate  our 
understanding  of  that  example  analysis. 


Probe  Topic 

Probes 

Goals 

What  were  your  specific  goals  at  the  time? 

Standard  Scenarios 

Does  this  case  fit  a  standard  or  typical  scenario? 

Does  it  fit  a  scenario  you  were  trained  to  deal  with? 

Analogues 

Did  this  case  remind  you  of  any  previous  case  or  experience? 

Hypotheses  & 
questions 

What  hypotheses  did  you  have? 

What  questions  were  raised  by  that  hypothesis? 

What  alternative  hypotheses  did  you  consider? 

What  questions  were  raised  by  that  alternative  hypothesis? 

Methods 

Were  you  following  a  method  you  had  learned  or  developed  in 
the  past  and,  if  so,  can  you  describe  it? 

Information  cues  for 
hypotheses  and 
questions 

As  you  co  llected  and  read  information,  what  things  triggered 
questions  or  hypotheses  that  you  later  followed  up? 

Information  tools 

What  sort  of  tools,  such  as  computer  applications,  did  you  use? 
What  information  source  did  you  use? 

What  difficulties  did  you  have? 

We  are  interested  in  performing  more  detailed  observations  of  analysts  as  they  solve 
some  tasks.  Our  general  idea  is  to  develop  a  set  of  tasks  and  related  materials  that 
we  could  present  to  analysts  to  solve.  We  would  then  record  how  they  solved  the 
task.  We  would  like  the  task  to  involve  the  collection  of  open  source  information, 
some  analysis,  and  some  product  such  as  a  briefing  or  paper.  We  would  like  to  see  if 
there  is  some  version  of  the  case  you  just  recalled  that  could  be  used  in  our  studies. 


Is  there  some  version  of  your  case  that  could  be  done  by  an  analyst  in  one  day  to  a  week? 


What  materials  would  be  needed? 
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Cognitive  Demands  for  Intelligence  Analysts 


•  Noticing  Data 

-  Time  critical  data  difficult  to  obtain 

-  Two  analysts  looking  at  same  picture/  image  can  see  different  things 

•  Need  background  knowledge,  expertise  working  with  each  type  of 
data  e.g.,  trends  of  country,  etc.  influence  interpretation 

-  Classify  data:  Relevant/  irrelevant 

-  Distilling  the  relevance:  “5  gems”  from  100  reports 

•  Synthesizing  Data 

-  Need  to  assimilate,  verify,  and  disseminate  in  short  time  window 

-  Combine  seemingly  unrelated  events  and  see  the  relevance 

-  Avoid  cognitive  biases 

•  Tunnel  vision:  So  focused  on  one  thing,  miss  key  piece  of  puzzle 

•  Confirmation  bias:  Discount/  ignore  data  don’t  understand, 
emphasize  type  of  data  most  experienced  with 

-  Prior  knowledge  used  to  assess  validity,  reliability 

-  Domain  expertise  needed  to  analyze  each  class  of  data 
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Cognitive  Demands  for  Intelligence  Analysts 

High  Cognitive  Workload:  GOAL:  Predict  complex,  dynamic  events 

-  Timeline  compression  coupled  with  organizational  constraints  “channels 
thinking  down  a  specific  path” 

-  High  cognitive  complexity  entailed  to  assess  reliability  and  synthesize  info  from 
wide  variety  of  sources:  Is  it  possible?  Plausible?  Other  sources  corroborate? 

-  Difficult  judgments  dealing  with  ambiguous  data 

•  All  types  of  data  can  include  spurious  signals,  inaccurate  information 

•  How  to  filter  and  how  much  weight  to  give  to  each  specific  piece? 

•  Domain  expertise  needed  to  analyze  each  type  (SIGINT,  HUMINT,  ELINT, 
IMAGERY,  MASINT,  Open  Source) 

•  Which  pieces  tie  together?  Can  I  trust  this  info? 

-  Overwhelmed  with  massive  amounts  of  data 

•  Need  to  resolve  discrepancies  across  systems,  databases,  etc. 

•  Need  to  interpret  data  in  context 

-  Time  pressure  becoming  increasingly  stressful  requirement 

•  “Incredible  time  crunch:”  Pressure  to  give  more  info,  and  faster,  to  the  customer 

-  Sheer  volume  makes  it  difficult  to  process;  yet  no  technology  available  to 
effectively  help  synthesize  different  types  of  info 

Cumbersome  Databases 

-  Poor  correlation  algorithms 

-  System  presents  results  users  do  not  trust:  “tracks  out  of  whack” 

-  Analysts  processes  data  manually  Adds  to  cognitive  load 
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Cognitive  Demands  for  Intelligence  Analysts 


Decisions  regarding  Information  Requirements 

-  Decide  whether  additional  data  is  required 

•  Who  to  go  to  for  specific  information? 

•  What  system  is  best  to  get  particular  types  of  information? 

•  Re-task  an  asset?  (expensive,  involves  trade-offs) 

•  How  to  be  efficient  with  limited  assets,  yet  many 
requirements? 

Disseminate 

-  Need  to  understand  customer,  context,  big  picture 

•  Don’t  overload  “customer”/  but  don’t  starve,  i.e.,  don’t 
decide  for  them  when  they  need  to  know  something 

•  Customer  doesn’t  articulate  need /  provide  reasons  for 
needing  info 

-  Avoid  Groupthink,  “Boy  who  Cried  Wolf” 

Cultural  Issues 

-  Credibility:  Operational  guys  “rarely  understand  analysis” 

•  Attitude:  “Why  do  I  need  you?” 
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Naval  Intelligence  Analyst  NPS2-2 


*  Task: 

-  Will  Estrada  be  deposed  from  presidency  of 
Philippine  Islands?  Will  there  be  a  coup? 

*  Deadline: 

-  6  weeks 

*  Time  worked  on  task: 

-  Student  estimate  72  h/week  x  6  weeks 

-  Our  estimate  300~400  h. 


NPS2-2:  Will  Estrada  be  Overthrown  in  a 

Military  Coup? 


Forage 

(search,  call,..) 


Extract, 
Summarize^ 
File 


A 


Timeline 


Decision 

table 


Scenarios 


INFORMATION  FORAGING: 
Dual  Problem  Space  Space 


(1)  Source  space 


AEmbassy~|c^: 


General 


Experts 


Press 


Websites 


JICPAC  materials 


Cables 


Travel  books 


Histories 


Cultural  ethnography 


East  West  Center 


DIA/DHS 


Asia  Pacif  ic  Center 


CIA 


A  Embassy 


US  War  College  Records 


(2)  Question  space 


Far  Eastern  Ec.  Rev  |  Questjons 


Past  gov  transformations?  | 


Key 

People 


Political 


Pro 
/  Estrada 

\  Anti 
Estrada 


Chief  of 
Staff 


Military 


Army 


Air  Force 


Senators 


Prev  coup 
plotters 

Organized 

crime 


Previous  plotters  now  senator 


Schemas 


•  KEY  PLAYERS 

MILITARY 
ARMY 
REGION  1 
<Commander> 
FAMILIES 

<Assistants> 
REGION  2 
<Commander> 
<Assistants> 

Logistics 

INTEL 
PERSONNEL 
AIR  FORCE 


POLITICAL 

SENATORS 

CLERGY 

PREY  COUP  PLOTTERS 


OTHERS 

ORGANIZED  CRIME 
PRESS 

PROMINENT 


■  ATTITUDES 

PRO-AQUINO 

ANTI-AQUINO 


CLIQUE  ASSOCIATIONS  ■  SOURCES 


SAME  UNIT 

SAME  REGION  OF  ORIGIN 
CLASSMATES 
FAMILTY  RELATIONSHIP 
PAST  CO-PLOTTER 
BOARD  CO-MEMBERSHIP 
BUSINESS  TIES 


GENERAL  LIT 

JICPAC 

CABLES 

WEBSITES 

EXPERTS 


POLITICAL  PARTIES  INVOLVED  SOME  WAY 
POLITICAL  ACTION  GRPS 
POLITICAL  FRONT  ORGS 
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Example 


Week  1 


Week  2 


Week  3 


Week  5 


Consult  old  material 
Background  research 
Consult  open  source  intelligence 


Collections  planning 


X 


yyeek  ^  ♦  Liaison  with  Intel  community 


X 


Build  brief/paper 


l 


Deliver  paper/brief 


Week  6 


Support  post-graduate  education  in  intelligence  analysis 
JITC  Briefing  toolkit 

Quickly  gather  info,  organize,  analyze,  and 

communicate  an  intel  briefing 

E.g.,  Observed  Chinese  naval  maneuvers  vs  stated 

doctrine 

E.g.,  Immediate  likelihood  of  unfriendly  military  coup  in 
the  Philippines 
Build  on 

Psychological  studies  of  intelligence  process 
Exemplary  human-information  interaction  technology 


Web  Behavior  Graphs  (WBGs) 

SI  S6  S7 


ANTZ 

sio 


CITY 

SI  S6  S7  SIO 


Summary:  Critical  Decision  Method 


•  CTA  was  conducted  to  support  development  of  a  model  of  the 
intelligence  analyst’s  processes,  biases,  and  analytic  strategies 

-  Goal:  Provide  input  to  build  a  computational  model  of  analysis 
process 

•  Needed  to  uncover  a  different  kind  of  domain  knowledge 

-  Critical  Decision  Method  was  tailored  to  focus  on  analysis 

-  How  an  Intelligence  Analyst  goes  about  the  task  of  analyzing  a 
complex  issue  of  strategic  concern 

•  Analysis  of  a  complex  cognitive  task,  such  the  intelligence  analyst's 
job,  often  requires  the  use  of  multiple  techniques. 

-  When  results  from  several  techniques  converge,  confidence  is 
increased  regarding  the  accuracy  of  the  CTA  model. 

-  Method  is  still  evolving 

-  Next  steps:  Additional  interviews  and  observe  analysts  perform 
task  using  open-source  information  on  www 
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